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ABSTRACT 

Bayesian estimation procedures are summarized and 
numerically illustrated by means of simulation methods. Procedures of 
data generation for simulation purposes are also delineated and 
computationally demonstrated. The logistic model basic to the 
Bayesian estimation procedures is shown to be explicit with respect 
to the probability distribution from tfhich one is sampling. This 
feature allows for an assessment or evaluation of its capabilities 
witb/out empirical data. The fit of the model to empirical data is 
discussed as an issue independent of considerations as to model 
capabilities. Three item banks are used to simulate Bayesian 
estimation procedures. Two are idealized — though reasonably 
possible — examples; whereas, the third consists of items specified 
according to other parameter estimates reported. Results indicate 
t&at with test validity held constant, Bayesian tailored testing of 
t/he Verbal Scholastic Aptitude Test (VSAT) could result in a savings 
of 65% of testing time for the average examinee. However, more 
savings in testing time is seen as possible through the use of item 
banks developed specifically for the purpose of tailored testing. The 
present investigation did not utilize prior information. Further 
assessments of model capabilities should explore such usage. While 
present results appear favorable, the full potentialities of the 
model have yet to be assessed. (Author/AE) 
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Bayesian estimation procedures derived by Owen (1969) were 
summarized and numerically illustrated by means of simulation 
methods. Procedures of data generation for simulation purposes were 
also delineated and computationally demonstrated. 

The logistic model basic to the Bayesian estimation procedures 
was shown to be explicit with respect to the probability distribution 
from which one is sampling. This feature of the model allows for an 
assessment or evaluation of its capabilities sans empirical data. 

The fit of the model to empirical data was discussed as an issue inde- 
pendent of considerations as to model capabilities. 

Three item banks were used to simulate Bayesian estimation 
procedures. Two of the banks were idealized- -though reasonably 
possible — examples; whereas, the third consisted of items specified 
according to parameter estimates reported by Lord (1968) for the VSAT. 

With test validity held constant, Bayesian tailored testing of 
the VSAT could result in a savings of 65$ of testing time for the 
average examinee. However, more savings in testing time was viewed 
as possible through the use of item banks developed specifically for 
the purpose of tailored testing. 

The present investigation did not utilize prior information. 
Further assessments of. model capabilities should explore such usage. 
While present results appear favorable, the full potentialities of 
the model have yet to be assessed. 
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Individualized Testing by Bayesian Estimation 



Owen (1969) has derived Bayesian procedures for the tailoring of tests 
for the cases where chance success on the items is and is not effective. 

Both cases will be discussed in the current report along with illustrative 
data.- Computer programs which simulate the process are described and 
included in the appendix for the separate cases. The programs can be modi- 
fied for "live" tailored testing applications. 

Under both cases,, the procedures: (l) identify the most appropriate 
item for presentation; (2) score the response to that item or, synonomously, 
(re-) estimate the ability parameter for the individual; and (3) calculate 
the standard error of the new estimate of ability. The process can be repeated 
until all or a specified number of items have been used or an allowable value 
of the standard error of estimate has been attained. 

In the following, we assume that the item parameters are either known 
or have been previously estimated. By way of review, the item parameters of 
the logistic model are item discriminatory power (a^); item difficulty (b^); 
and. probability of chance success on the item (c^). Methods of estimating 
the item parameters have been discussed by Birnbaum (1968). 



Method 



In both cases, one calculates §, y the estimate of ability, and 
0/ the variance of the estimate, sequentially. The subscript p indexes 
the number of items that have been presented to an individual during an evaula- 
tion sequence. For example, if one lacks prior information on the individual 
being examined, ®(q) ant ^ a (0) wou -^ he se ^ & h values of 0 and 1, respec- 
tively. Initial values of this nature have the basic rationale that in the 
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absence of individual information, the mean, 0, is the most probable estimate 
of ability, ®(q) ’ w kile standard error, a (o) J c °i nc i <,es with the 
standard deviation. More concisely expressed, the prior distribution of 
ability, 0 , is assumed to be N(0,l). 

Case 1^ Chance success on the items is not effective 

In order to determine the item most appropriate for immediate 
presentation, we calculate for all (unused) items. The formula is 



(1) a. = [a~ 2 + 3^3 ex P (2D^)[l - (erf D.) 2 ] 

where 



( 2 ) 



D. 

1 




9 (p) ).(2 [ a- 2 + 




and 

(3) erf D. = — 

1 •/« 

More familiarly, we have 



D. 

t p 

/ exp(-t ) dt 

0 



(k) erf D i = 2c&( \[2D.) -1 



where 

( 5 ) 






*f\ 2D. 



-CD 



exp [~^ ] dt 



is the well tabled normal probability function. For future reference and 
convenience, we will designate the following: 



(6) 



S. 

1 



-2 

°(p) 




"2" 

(p) 
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(7) 


t i = [l + 


-2 «-2 ,-1 
a . Of] 

i (p; 


(8) 


u - (i - 


erf D. ) 

l 


(9) 


v. = (l + 
1 


erf D. ) 

l 



Nov; the subscript, i, of the smallest at given by equations (l) 

"fch 

identifies the optimal item for presentation. Upon presentation of the i — 
item, one of two outcomes is possible. The individual will get the item right, 
or he will get the item wrong. Given that he responded correctly, his new 
ability estimate is given by 

(10) ®(i*i) (rigM;) ‘ ®<p) + V'f vl 1 

The standard error of the estimate, conditional, upon a; correct, response, is 
given by the square root of 

(H) °fct-l)( right ) = °^p) (1 " 7” t if ex P ( D i^ u i^ 

V It 

• [ — - D. exp (D 2 )u. ] } 

/ — 1 i i 

v n 

Should the individual miss the item, his new ability estimate is given by 

< 12 > Vi) ( ™« ) = S (p)-/7 a,( 4'i , i 1 • 

The standard error of the new estimate of ability, conditional upon an 
incorrect response, is given by the square root of 

(13) o? )(wrong) = o^{l ~ ~f~ t ± [exp 

V It 

•[ + D i exp(D 2 )v i ]} 

V Tl 
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At this point, the number of item presentations (p) is updated by 
one since the sequential use of equations (l) and equations (10) and (ll) or 
equations (12) and (13) defines an item presentation cycle. As a consequence, 

£ ^2 A /s2 

the current 0/ ilN and O/ . % become the 0/ x and a/ \ when a new 
(p+i) (p+i) (p) (p) 

item presentation cycle is initiated.. When equations (l) are recalculated 

th 

for the (n - p) unused items, the i — item is again identified by the 
smallest (X . Again, depending on the propriety of the individual's response 
either equations (10) and (ll) or equations (12) and (13) are used. The 
cycles may be repeated until a termination criterion has been attained. 

Case II: Chance success on the items is effective 

£ 

Equations analogous or even identical to those for Case I exist for the 
present case; however, due to the effectiveness of guessing, some equations 
increase in complexity. Let us designate 



U*) 



(1 - c )u 

w. = [c. + ^ ] 



for further reference. Now one begins by calculating 

( 15 ) P ± « [(1 - c i )t i ]" 1 w i (1 |)exp (2D£)[l..+ c i (l 1 )]~ 1 , 

for all (unused) items where equations (2) through ( 9 ) still obtain. Again 
the subscript, i, of the smallest |3^ identifies the optimal item for 
immediate presentation. 

bh 

One of two outcomes will occur when the i — item is presented. If 
the individual responds correctly, his new estimate of ability is given by 



( 16 ) 



Vu (right) ■ %) + 



(l ~ c i ) 2 

exp(-D.)s.w. 

^ 1 ’ ’ 



-1 
± i 
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The standard error of the new estimate, conditional upon the appropriate 
response, .is given by the square root of 






(17) °(p + l) (ri 6 ht) * ®(p) 



(1 - c.) o 

1 (w.u. ) t. exp(-2D.) 



Sn 



i i 



([ — -r- " Dj exp(D?)u i l - w" 1 } 

V Jt V It 



Should the individual respond incorrectly to the i — item, equations (12 ) 
and (13) are still appropriate; however, for convenience the equations are 
repeated. His new estimate of ability is provided by 



( 18 ) 



Vl) (l ' ron S ) = 6 (p) 




f -1 

exp(-D i )s i v i 



The standard error of the new ability estimate, given a wrong answer, is 
obtained from the square root of 



(19) o^i) (wrong) = o? v (l - -y- t ± [exp (D?)^]" 2 

v n 



• [ + jl). exp(D. )v. ]} 

r 1 1' 1 

v u 

The sequential process of item presentation cycles delineated above is 
also appropriate here. The optimal item for immediate presentation from 
among the (n •• p) unused items is determined for each cycle by the subscript 
on the smallest as given by equations (15)* The i — item is responded 
to by the individual and the nature of his response determines whether 
equations (l6) and (17) or (l8) and (19) are to be used in estimating ability 
and the variance of the estimate of ability. As indicated above, termination 
criteria may be specified on the basis of a maximum value for p and a 
maximum allowable value for d/ \ . 
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’(p.1) acd °(jh 1) are the first 



Under both cases, the estimates 0^ 
and second moments of the posterio r distribution . When these same estimates 
are used to determine the next item to be presented their status changes to 

A ^2 

9/ v and a , \ , the first and second moments of the prior distribution. 

In Bayesian estimation procedures, the posterior distribution becomes the 
prior distribution when a new item presentation cycle is initiated. 
Generation of response vectors for simulated individualized testing 



Given any set of n items having known item parameters, the probability 
distribution, conditional upon ability, 9, can be determined for all pos- 
sible response patterns or vectors. What this means is that if one has a 
random sample of values of 9, one can then sample response vectors and 
score these by the procedural manner he chooses- Later we will discuss how 
one can evaluate the given procedure. In this instance, we choose to eval- 
uate Bayesian estimation procedures, but the simulation technique has wic'er 
applicability. A case in point would be an evaluation of flexilevel testing 
(Lord, 1971). 

For purposes of concrete illustration, we will take a 4-item example. 

Given four items, there are 2^ or 16 possible patterns or response vectors, 

v, . These are: 
k 



v.. = [0 0 0 0] 

x 

v 2 =[0001] 
v^ =[0010] 
v^ = [001 1] 
v r = [010 0] 

5 

Vg =[0101] 

v ? =[0110] 

Vg =[0111] 

v 9 =[1000] 
v 1Q = [1001] 
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v ±1 - [1010] 

v 12 = [1011] 

v 15 - [1 1 0 0]_ 

= [110 1 ] 

v. _ = [1110] 

-L? 

V /- = [ 1111 ] 

lo 

Now response vector indicates that items 1 , 2 , 3 > and 4 were responded 

to correctly, incorrectly, correctly and incorrectly, respectively. If, 
further, the item parameters for the items of illustration are: 





a. 

l 


b. 


c . 
i 


Item 1 


1.6 


1.1 


.06 


Item 2 


2.0 


1.1 


.05 


Item 3 


1.2 


1.1 


.05 


Item 4 


1.0 


1.1 


•13 



and, say, the value of 0 for which one is evaluating the probability 

distribution of the v. is 1 . 0 , we would proceed as follows.. The probability 

K 

of a correct response to an item, given 0, is provided by the model as 

(20) P i ( 0 ) = c i + (1 " c j.) 1 + expl-Da^O - b i )J 

r - 

where D is the constant 1 . 7 * The probability of missing an item is merely 



( 21 ) Q.( 0 ) = 1 - P.(©) 




As a consequence, the following probabilities obtain: 



P 1 (e = l.o) = 


.47 


Q x (0 = 1 . 0 ) = 


•53 


P 2 (0 = 1 . 0 ) = 


.45 


^(0 = 1.0) = 


•55 


P ? (0 = 1 . 0 ) = 


.48 


Q 5 (0 = 1.0) = 


•52 


P 4 (0 = 1 . 0 ) = 


•53 


Q 4 (0 = 1.0) = 


•47 
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Since ability is fixed, the probabilities are assumed independent across 
items. This assumption is more familiarly known as that of local independence . 
It merely states that if several variables covary with one another due to a 
distinct variable, holding the latter constant results in independence among 
the several. We may now compute the joint probability of the independent 
events indicated by each v for 0 equal to 1.0. This is merely the 
product of the probabilities of the events recorded by the zeros and ones 
in each v. . For convenience, the probabilities conditional upon 0 are 
calculated in Table 1. For example, response vector v^ recorded that: 

Item 1 was correct with probability equal to P^(0 = 1.0), or .47; Item 2 
was incorrect with probability equal to Qg(® - 1*0), or .55; Item 5 was 
correct with probability equal to P^(0 = 1.0), or .48; and Item 4 was 
incorrect with probability equal to 0,^(0 = 1.0) or .47* The probability of 
the joint events conditional upon 0 = 1.0 is, then, (. 47) ( *55 ) (• 48) (.47) 
or .O 5 O 3 . In Table 1, the cumulative conditional probabilities are also 
given. Handily, as well as properly, these sum to unity. As a consequence, 
one can obtain a randomly selected response vector for a given value of 0 
by obtaining a random number from a distribution which is uniform on the 
interval from zero to unity and comparing this to the attendant probability 
intervals. For the sake of clarification, let us say that a random number 
thusly selected was . 6254 . Since the value occurs in the probability inter- 
val of -5933 to .6644 corresponding to response vector v^ , the said 
vector would have been randomly selected in proportion to its probable 
occurrence given the stated conditions. Clearly, the nature of the sampling 
remains unchanged even with an arbitrary ordering of the response vectors. 
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Table 1 



Conditional Probabilities Cumulative Conditional Probabilities, 



and Probability Intervals for Possible Response Vectors 



Conditional Probabilities 



Prob (v^ 


l e - 


1.0) = 


( 


53) (-55 )( 


52)(-47) 


Prob (v^ 


l e = 


1.0) = 


( 


53) (-55 ) ( 


52 ) (-53 ) 


Prob (v^ 


l e - 


1.0) = 


( 


53) (-55 ) ( 


^8)(.U7) 


Prob (v^ 


|e - 


1.0) - 


( 


53 ) ( - 55 ) ( 


U8)(.53) 


Prob (v.. 
3 


l e = 


1.0) a 


( 


53)(-U5)( 


52) (-47) 


Prob (vg 


l e - 


1.0) a 


( 


53)(.U5)( 


52) (-53) 


Prob (v^ 


|e = 


1.0) a 


( 


53) (-^5 ) ( 


48) (- 47) 


Prob (vg 


l e = 


1.0) = 


( 


53)(.U5)( 


48)(. 5 3) 


Prob (v^ | 


6 = 


1.0) a 


( 


^7) (-55 ) ( 


52) ( . 47 ) 


Prob ( Vpol 


e = 


1.0) = 


( 


^7) (-55 ) ( 


52 ) ( - 53 ) 


Prob(v xl | 


e = 


1.0) a 


( 


V7)(-55)( 


48) (. 47 ) 


Prob(v 12 l 


e = 


1.0) = 


( 


^7) (• 55 ) ( 


48)(-53) 


Prob (v^ | 


1 e = 


1.0) = 


( 


V 7 )(- 45 )( 


52) (- 47 ) 


Prob (v^ 


e = 


1.0) = 


( 


V 7 )(.U 5 )( 


52 ) ( . 53) 


Prob (v | 


e = 


1.0) = 


( 


4 f)(. 45 )( 


48 ) (- 47 ) 


Prob (v x g | 


e = 


1.0) = 


( 


^ 7 )(-^ 5 )(-W)(. 53 ) 





Cumulative 

Conditional 

Probabilities 


Probability 

Intervals 


.0712 


.0712 


.0000 to .0712 


.0803 


.1515 


.0713 to .1515 


.0658 


•2173 


.1516 to .2173 


.0742 


.2915 


.2174 to .2915 


.0583 


.3498 


.2916 to .3498 


.0657 


.4155 


.3499 to .4155 


.0538 


.4693 


.4156 to .4693 


.0607 


• 5300 


.4694 to .5300 


.0632 


•5932 


.5301 to .5932 


.0712 


.6644 


• 5933 to .6644 


.0583 


.7227 


.6645 to .7227 


.0658 


.7885 


.7228 to .7885 


.0517 


.8402 


.7886 to .8402 


.0583 


.8985 


.8403 to .8985 


.0477 


.9462 


.8986 to .9462 


.0538 


1.0000 


.9463 to 1.0000 
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The principles developed above generalize directly to situations where 
there are a large number of items; however, it is then usually more conven- 
ient to work with repeated samplings from subsets of items with 0 fixed. 
Notice that with 100 items the calculation of 2^^ conditional probabili- 
ties for fixed 0 would present severe computational problems. 

A computer program that accomplishes the sampling of response vectors 
has been presented elsewhere (Urry, 1970)- I lore specifically, the program 
samples response vectors for a random sample from the assumed distribution 
of 0, N(0,l). Obviously, gaussian random numbers will fulfill the imposed 

sampling requirements with respect to underlying ability. 

Notice that only a subset of items from any response vector of length 
n are actually used by the Bayesian procedures to obtain an estimate of 



ability, 0 



(p) 



The item sequence, as noted above, is determined by prior 



information and/or responses as well as the item parameters. Given the model, 
the temporality of responses, as far as the n-length response vector is con- 



cerned, is inconsequential. In other words, one may (re-) estimate ability 
on any sequence or sub^t^of items from the response vector as determined by 
a procedure while ignoring tfte x available responses to the remaining items. 
The simulated or after-the-fact tailoring of empirically obtained response 



vectors is also possible. 

Evaluation of the Bayesian Procedures 

/s 

To evaluate the procedures, one merely correlates the 0/ \ against 
the 0 for the randomly sampled "cases" or simulated individuals for a 
particular termination criterion. Underlying ability, 0 , is the perfect 
criterion for the computation of this validity coefficient- 
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Regarding valid evaluation, the specifications for the item parameters 
of an item bank are. critical. For example, high item discriminatory powers 
and a rectangular distribution of item difficulties have in other tailored 
testing contexts led to quite satisfactory validities. Here we will consider 
three item banks, two of which are idealized while the third is taken from an 
empirical source. 

Item Bank A consisted of 100 items. The item discriminatory powers, 
a^ , equalled 1.6 for all i, i = 1, 2, ... n or 100, while 20 items each had 
item difficulties, b^, at one of five levels, i.e., -I.50, -.75; *00, .75; 
and 1.50, respectively. The probability of chance success on the items, c^ , 
was .2 for all items. 

Item Bank B consisted of 105 items. Again, the item discriminatory 
powers, , equalled 1.6 for 8.11 items, while 5 items each had item diffi- 

culties at one of twenty-one levels, i.e., -2.50, -2.25, -2.00 ... 2.5O, 
respectively. The probability of chance success on the items, c^ , was, 
again, .2 for all items. 

The item parameters for Item Bank 0 were taken from estimates provided 
by Lord (1968) for the Verbal Scholastic Achievement Test (VSAT). The 
estimation sample was comprised of 2,862 cases. The interested reader will 
find the specifications for this item bank enumerated in that source. 

With these specifications, simulated response vectors were generated, 
as previously outlined, for samples of 50 each for Item Banks A and B. For 
Item Bank C, 100 "cases" were generated. Termination criteria for Item Banks 
A and B were set at .52 and .25 as maximum allowable values for a/ ^ • Under 



the ’ termination— rules-, — p — wi H— vary— a crOiS s~~s imula t ed ex a m ine e s..«_ For Item. 



Bank C, 



the compound termination criterion was 



p = 50' or o (p) 



equal to or 



d 
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less than a maximum value of .25. 
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Results 

In order to illustrate what an individual evaluation sequence would look 
like, Table 2 is provided. The data are for a simulated "case" who was eval- 



uated with Item Bank A. At the beginning of evaluation, nothing was assumed 
known about Examinee 1046; therefore, our prior information admits to this 

A ^ 

state of ignorance by setting ®(o) ant ^ °(0) an d 1*00, respectively. 

In other words, the. mean is, then, the most probable value or the best esti- 
mate of 0 while the standard error of the estimate corresponds to the 
standard deviation of the prior distribution. Given this initial information, 
Item 3 from Item Bank A was found via equations (15) to be the most appropriate 
for the first presentation. The examinee answered Item 3 incorrectly, so that 
equations (l8) and (19) were used to calculate the specific values of -.6766 

A A 

and .7363, respectively for and °(i) * Using these estimates, equations 

(15) were evaluated for the (n - l) or remaining 99 items. Item 2 from Item 
Bank A was, thereby, identified as the most appropriate item for the second 
presentation. The examinee answered Item 2 correctly; consequently, equations 

(16) and (17) were used to calculate ®(2) and °(2) ; or “'3904 and .6694, 
specifically and respectively. The cyclic processing continues, as indicated 
earlier, until a termination criterion is reached. At a termination criterion 

A /s 

of less than or equal to .32, 0 (io) or "*9937 was the Bayesian esti- 



mate of ability for Examinee 1046. Given the termination criterion of a 



(p) 



/A 

less than or equal to .25, ^(14) or “1-2104 was the Bayesian estimate of 
ability. Now the "true" value of 0 was -1.2180 for the particular examinee. 
Notice that if we establish confidence intervals for 0 with an approximate 
^probability of • 93 ) nave: 

Prob [-.99 - (l-96)(-32) < 0 < -.99 + (1.96)(.32)] = -95 
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Table 2 

An Example of an Individual Evaluation Sequence by- 
Bayes ian Estimation Procedures 



Examinee Number 1016 



Prior Information 



e ( or * 00 



'( 0 ) 



= 1.00 



Item 


Item Bank 


Item 


Ability 


Standard : 


mentation 


(p ) Number 


Response 


/s 

Estimate [0^ j] 


of Estimate 


1 


3 


0 (wrong) 


- . 6 766 


.7363 


2 


2 


1 (right) 


-.3901 


• 66 9*'- 


3 


7 


1 (right) 


- . 1902 


.6079 


1 


8 


0 (wrong ) 


-.1718 


.51I0 


5 


12 


1 (right) 


-•3330 


.I789 


6 


17 


0 (wrong) 


-.6717 


.I085 


7 


22 


1 (right) 


-•5597 


.3895 


8 


27 


0 (wrong) 


-•7591 


• 3500 


9 


32 


0 (wrong ) 


-.89II 


• 3221 


10 


37 


0 (wrong) 


-•9937 


.3018 


ll 


1 


0 (wrong) 


-1.1660 


.2791 


12 


6 


1 (right) 


-1.1230 


.271I 


13 


11 


0 (wrong) 


-1.2506 


.25I9 


ll 


16 


1 (right) 


-1.2101 


.2189 



(p) 
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or 

Prob [-1.62 < 9 < - -36] = .95 

and 

Prob [-1.21 - (I. 96 ) (-25) < 0 < - 1.21 •» (I. 96 ) (. 25 )] 2 .95 
or 

Prob [-1.70 < 9 < - . 72 ] 2 -95 

where 9 is well within the indicated intervals. The probabilities are 
approximate since the f, ^ are unbiased only when p is large. 

We now turn to results in samples where the individualized testing 
delineated above was applied to each "case." Validity coefficients for the 
three item banks are presented in Table 3* For example, using Item Bank A 
with a termination criterion of o/ \ equal to or less than - 32 , the validity 
coefficient for a sample size of 50 was . 928 . In attaining the termination 
criterion, the minimum, average, and maximum number of items used in evalua- 
tion were 8 , 12.2, and l 6 , respectively. Analogous interpretations apply to 
the remaining rows of data in the table. In the simulation of the tailoring 
of the VSAT, Item Bank C_, it was found that the validity coefficient for the 
80-item raw total score was .94-9; which was the same as that reported in Table 
3 for tailored tests of an average length of 27.6 items. On the average, 65 % 
of the original test may be considered unnecessary for examinees to take, since 
comparable validity in Bayesian tailored tests can be obtained with this sub- 
stantial reduction in test length or items used. 



Discussion 

In t he parl ance of the fanto r. analyst., the v alidi ty ongff i Pi pnt. used here 

is similar to a factor structure coefficient or the correlation between the 
variable and the factor of ability. In the present context, the novel usage 





15 



Table 3 

Validity Coefficients and Pertinent Data for Bayesian 
Tailored Testing with Three Item Banks 



Item 


Termination 


Number of Items 


Validity 


Sample 


Bank 


Criterion 


Minimum 


Average 


Maximum ... 


Coefficient 


Size 


A 


*(P) * 02 


3 


12.2 


16 


.928 


50 




LT\ 

OJ 

VI 

3 

<D 


lk 


18. k 


22 


.956 


50 


B 


OJ 

n~\ 

VI 

3 

<o 


8 


11.5 


lk 


r- 

OJ 

ON 


50 




°(p) 2 


12 


17- k 


21 


• 9k8 


50 


C 


S (P) S 














or p = 30 


lk 


27.6 


30 


•9k9 


100 
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of the word "variable" should not obscure the direct analogy. However, there 
is an important methodological distinction to be made. The logistic model 
allows one to avoid difficulty factors. For example, if we were to intercorre- 
late the items from any of the item banks — A, B, or C — the correlation matrix 
would form a simplex. In a simplex, the magnitudes of the correlations between 



I . i • xn xacuor analyzing a matrix of this nature, a plurality of 

' J 

factors is possible even though the basic source of the data is explicitly 
unidimensional. The model, then, circumvents a problem in factor analysis 
that has been viewed by several investigators (Ferguson, 19^1; Gibson, 196O; 
Green, 195 2 ) with some concern . 

If the logistic model fits empirical data, the inferences made from 
simulation studies are applicable to those empirical situations where the 
constituent items of an item bank correspond in terms of their parameters to 
those that have been simulated. As seen above, one may fix ability by choosing 
a person at random or by selecting a gaussian random number. As far as the 
model is concerned, the operations are identical since underlying ability is 
assumed to be N(0,l). Thereafter, the model maices explicit the probability 
distributions from which one samples the response vectors for items of speci- 
fied parameters. The upshot is that, if the model obtains for empirical data, 
simulated random samples do not differ in any critical way from empirical 
random samples. 

Obviously, one can assess model capabilities independent of the 
determination of the fit of the model to empirical data. The question of 
model capabilities is, therefore, the more basic since, if the model does not 
show sufficient promise, tests of empirical fit are superfluous. Given 




as the disparities in the item difficulties 
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sufficient promise, the question of empirical fit, then, becomes important. 
Methods of assessing the fit of the model to empirical data are discussed by 
Bimbaum (1968). While the results of this investigation would underscore 
the importance of the methods, no attempt will be made here to examine them. 

Wow Item Banks A and B were presented as idealized examples; but, if 
one has the specific objective of tailoring tests, reasonably similar banks 
might be achieved in practice. On the other hand, Item Bank C has (admitting 
to minimal errors in estimation due to the large sample size) its counterpart 
in the form of an extant conventional test, the VSAT. The findings with 
regard to Item Bank C indicated that if the VSAT were used in Bayesian 
tailored testing applications, 65$ of the test would be unnecessary for the 
average examinee to take. However, if we look at the results for Item Banks 
A and B, an average of 27.6 items in relation to 18.4 and 17.4 items for 
evaluation with comparable validity shows that there is room for improvement. 
The improvement would be realized by designing item banks for the specific 
purpose of tailored testing. 

The VSAT was designed for the specific purpose of univariate selection 
where the selection ratio is low. In other words, the test was constructed to 
minimize the errors of measurement in the range of high scores. To accomplish 
this purpose, items of higher than average difficulty were more frequently 
selected to comprise the test. In a tailored testing context, however, it is 
advisable to have a distribution of item difficulties which extends uniformly 
through the full range of difficulty (Urry, 1970). The specific purpose of 
tailored testing, then, is best served by a different item selection technique 
than that used in the construction of the VSAT. 
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Notice, further, that we have not utilized the Bayesian estimation 
procedures to the fullest extent. In most testing applications we begin 
under the tacit assumption that we know nothing about the examinees. In the 
majority of cases, this state of ignorance need not be assumed. Further 
assessments of model capabilities in regard to Bayesian estimation procedures 
should explore this attractive feature. The possibility exists that evalua- 
tion sequences could be further reduced in terns of average number of items 
while a satisfactory level of validity is maintained. 
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Appendix 



The order and preparation of the input specific to the computer programs 

are described here. Also, source listings are provided for both programs. 

Case I, where chance success is not effective, corresponds to program BAZES2. 

The particular designation is used to indicate that the items used in the 

Bayesian estimation of ability have but two parameters since the ^ are null. 

Similarly, program BAZESJ corresponds to Case II where chance success on the 

items is effective or the c. are non-null. 

1 

The author wishes to acknowledge the able assistance of Jerry W. Edwards 
of the Bureau of Testing. :ds programming of FUNCTION ERF, which is used in 
both programs, was of considerable aid. 
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Order and Preparation of Cards Specific to the Programs 



( 1 ) 



( 2 ) 



Title Card (Format 200) 
TITLE Col. 1-70 

NVF Col. 75-76 

NCASES Col. 77-80 

Problem Card (Format fijk) 
NAME Col. 1-40 

ITERM Col. 64 



IUSE Col. 67-68 



EPSILON 



Col. 69-76 



Any alphanumeric title for the identification 
of individual evaluation sequences. 

Number of variable format cards. 

Maximum: 5 

Number of individuals to be evaluated. 

Name of the item bank. 

Specification of termination of criteria. 

Indicate: 

1 If the evaluation sequences are to be 
terminated after a given number of item 
presentations. 

2 If the evaluation sequences are to be 
terminated after an allowable value of the 
standard error of estimate has been attained. 

5 If either of the conditions stated for 
1 or 2 obtain. 

Maximum number of items to be presented. 

Must be specified if Col. 6k contains a 1 or 5 . 
Cannot exceed the number of items in the bank. 
Maximum allowable value for the standard error of 
estimate. Use F8.4. Must be specified if Col. 64 
contains a 2 or 5 . Suggested range: .4 to .2. 

Number of items in the item bank. Maximum: 200. 





NITEMS Col. 77 - 8 O 



(5) Item Parameter Cards (Format ft 31) 

(a) Item discriminatory powers (a^) 

Ten per card: 

Col. 1-8 a F8.it 

Col. 9-16 a 2 F8.it- 

Col. 73-80 a F8.it 

Repeat on the required number of cards. Maximum: 200 parameters 

or 20 cards . 

(b) Item difficulties (b^) 

Ten per card: 

Col. 1-8 b F8.it- 

Col. 9-16 b 2 F8.it 

Col. 73-80 b F8. it- 

Repeat on the required number of cards . Maximum: 200 parameters 

or 20 cards. 

Necessary only for program RAZES3 

(c) Probability of chance success on the items (c. ) 

1 

Ten per card: 

Col. 1-8 c F8.it 

Col. 9-16 c 2 F8.it 

Col. 73-30 c F8.it 

Repeat on the required number of cards. Max imum : 200 parameters 



or 20 cards. 




(4) Variable Format Card(s) 



A maximum of five cards can be used to describe the data. Use columns 
1-80. Funch a regular FORTRAN format statement omitting the word 
"FORMAT." Begin with "(l4," to accommodate an individual identifica- 
tion number, and use "II" to input each binary response. 

(5) , Input Data (Variable Format) 

\ 

j The input data consist of an identification number and binary 
! responses for each case as described on the variable format card(s). 
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PROGRAM BA7.ES2 ( INPUT * OUTPUT, TAPF5= INPUT* TAPE 6 '^OUTPUT ) 

DIMENSION A (200) »B(200) , 01200 ) , ALPHA (200) ♦ I R I 200 ) ♦ I EL IM ( 200 ) * 

X I SUB (200) »R(200) ,ER( 200 > , FMT < 100 ) * T I T LE ( 7 ) , NAME! 4) 

P 1*0.143 592653589793 

Cl = 1 « 0 /SORT (PI) 

C 2 = 2 • 0 /SORT (PI) 

C 3 = SORT < 2.0/PI ) 

SIGN = -1.0 

200 R EAD ( 5 » 201 ) (TITLE! 1 > * I s 1 » 7 ) *NVF ♦ NCASES 

201 FORMAT ( 7A10 , 2X , 2 I 4 ) 

1 F ( EOF » 5 ) 300,301 

301 WRITE (6. 70) 

70 FORMAT (lHl »//////////////) 

WRITE (6, 71 ) 

71 FORMAT ( I HO * 47X » 3 5HBAYES I AN ABILITY ESTIMATION PR06RAM/54X, 

X23HTW0 ITEM PARAMETER CASE//////) 

WRITE ( 6, 72 ) 

72 FORMAT! 1H0 ♦ 36X » 1 7HBUREAU OF TESTING /////) 

WRITE(6»73) 

73 FORMAT ( 1H'-' * 52X » 24HUN I VERS I TY OF WASHINGTON ////////) 

R EAD ( 5*74} NAME , I TERM ♦ I USE * EPS I LON »N I TEMS 

74 FORMAT ( 4 A 1 0 ♦ 2 2 X » I 2 * I4,F3.4,I4) 

W R I T E ( 6 » 7 5 } N AM E » N I T E KS 

75 FORMAT ( 1H0 » 2 5X * 3 OH I TEM BANK ♦ 5X * 4A10 / / / /26X » 12HT0TAL ITEMS *4Xt 

X I 4/ / / / ) 

I F( I TERM-2) 76, 77? 73 

76 EPS I LON* 

GO TO 79 

77 TUSF=MTTFMS 

78 CONTI NUF 

79 WRITE(6»80) IUSE»EPSILON 

80 FORMAT! 1HO,25X,20HTERMINAT ION CR I TER I A/ /30X , 1 6HNUMBER OF ITEMS , 

XI 10//30X, 15HSTAN0ARD ERROR ,3X,F10.4) 

READ (5,31) ( A ( I ) ,1 = 1, N ITEMS) 

READ (5,31) (B(I)»I-1»N ITEMS > 

31 FORMAT (10F8„ 4) 

WR I TE ( 6 ♦ 84 ) 

84 FORMAT (1 Hi) 

CALL VARFMT ( NVF*FMT ) 

NK •- 0 

100 NK = MK + 1 
IPRES * 0 

IF (NK-NCASFS) 101,101,200 

101 WRITE (6,28) (TITLE! I ) .» I = 1 » 7 ) 

28 FORMAT ( lHl »30X»7A10////) 

RE AD (5, FMT) I D , ( I R ( I ) » I =1 ,N ITEMS ) 

ABLE = 0 * 0 
VAR--: 1.0 
SEE=SQRT ( VAR ) 

WRJTE(6»01) I D » ABLE s VAR 

81 FORMAT ( 1H0 * 10X»15HEXAMINE£ NUMBE R » 1 10/ / 1 6X * 1 5HPR I OR EST IMATES//21 X 
X » 8HAB I L I TY , F 10 . 5/ 2 IX , 9HVAR I ANCE ,F9.5//) 

WRI TE ( 6* 82 ) 

82 FORMAT ( 1H0 , 1 5X , 4H I TEM » 24X * 9H I TEM BANK , 24X , 7HAB I L I T Y , 2 2X » 

X14HST ANDARD ERROR /l 2X , 1 2HPR E5ENT AT I ON , 2 1 X , 6HNUMBER , 26X , 8HEST I MATE , 
X22X.11HOF ESTIMATE//) 
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DO 20 I = 1 »N ITEMS 
I EL INK I ) 0 

NLEFT " 0 

IFUPRES*G£. IUSE) GO TO 100 

IF ( EPSI LON<,GC • SFF ) GO TO 100 

DO 13 1*1* N ITEMS 

IF ( I - 1 EL I M ( I ) ) 12*13*12 

NLEFT *- NLEFT + 1 

I SUB (NLEFT) - I 

CONTINUE 

IF ( NLEFT ) 100.100*14 
DO 10 I si, NLEFT 
ITEM = I SUB I I ) 

R ( 1 ) = 1 » 0 / ( A ( I TEM ) * A ( ITEM) ) + VAR 
DU) = ( 3 ( ITEM) -ABLE) / SORT ( 2»0*R (I ) ) 

ER( I ) =ERF (DU ) ) 

ALPHA! I) “ R( I)*EXP( 2 . 0 *DU ) *D( I n*(l*0-ER!I)*ER( I») 

RMIN = ALPHA! 1) 

JSUB = 1 

DO 1 J~1 *MLEFT 

I Ft RMIN - ALPHA! J)) 1>1»2 

RMIN * ALPHA ( J ) 

JSUB = J 
■'CONTINUE 

ITEM = I SUB (JSUB) 

RITEM * R(JSUB) 

DITEM = D ( JSUB ) 

IELIM (ITEM) = ITEM 
IPRES = IPRES + 1 
TEM ~ DITEM*DITEM 

FI = C2*( 1.0/(1.0+( 1.0/(A(ITEM)*A( ITEM) }*! 1.0/VAR) )) ) 

I F ( I R ( I TEM ) ) 3, 3, A 
PER « 1 # 0 + ER ( JSUB ) 

ABLE - ABLE-C3* (VAR/SQRT (R ITEM) } *EXP ( S I GN*TEM ) /PER 
F 2 = l,C/( (EXP(TEM)*PER)**2) 

F 3 = C1+DITEM*EXP(TEM)*PER 
GO TO 5 

SER = 1 « 0~ER (JSUB ) 

ABLE « ABLE+C3* ( VAR/SQRT (RITEM) ) *£XP ( SIGN*TEM ) /SER 
F2 = leO/( (EXP!TEM)*SER> **2 > 

F3 = Cl-D I T EM* EXP { T EM ) *SER 
VAR a VAR *U..0-F1*F2*F3) 

SEE=SQRT ( VAR ) 

WR’I TE ( 6 s 83 ) IPRES, ITEM, ABLE .SEE 

FORMAT! 1H0,15X»I4*26X» I 4 » 26X , F8 , 4 » 25X » F8. 4 ) 

GO TO 50 

STOP 

END 

FUNCTION ERF(X) 

ERROR FUNCTION FOR) 0 I X - 3.95, MACCLAURIN SERIES 
FOR X * 10', ERF (X) = 1*0 
ACCURACY PARAMETER IS .IE-10 

ERF=0.0$lF(X.EG.0.0)RETURN$ES=SIGN( 1.0*X)$XX=E=Y=ABS(X) $J=1SIF (Y.L 
XT * 3 • 96 ) GOT 01SERF=1» 0 SR E TURN 
S=F=1$D02I=1»60$S=S*(-1 ) iF=F#I$J=J+2$XX=XX*Y*Y 
T=XX/( J#F)SIF!T.LT.. IE- 10 )G0T03$E=E+T*S 



2 CONTINUE 

3 ERF=A8S(E )*2«0/l .772453 8 509 055 2*ES$RETURN$END 
SUBROUTINE VARFMT ( NVF » FMT ) 

NVF= NUMBER OF VARIABLE FORMAT CARDS 
DIMENSION FMT ( 1 ) 

NVF-NVF*20 

R EAD ( 5 *47-M ( FMT ( I ) » I = 1 »NVF ) 

470 F ORMAT ( 20A4 ) 

WRITE (6 *471 ) (FMT (I) » 1 = 1. NVF) 

471 FORMAT ( 1H0 * 7H FORMAT/ ( 10X »20A4 ) ) 

RETURN 

END 



PROGRAM 0A7.ES 3 ( INPUT » OUTPUT* TAPE5=INPUT» TAPE6=OUTPUT ) 

DIMENSION A ( 200 ) » B ( 200 ) >C ( 200 ) ,D { 200 ) »BET A { 200 ) * I R < 200 ) » 

X I SUB l 200) >R( 200) »ER( 200 ) »FMT I 100) .TITLE (7 ) * NAME I A ) ♦ 

X IF.UMI200) 

P 1=3.141592653589793 
Cl = 1 . O/SQRT (PI) 

C2 = 2 o O/SQRT ( P 1 ) 

C 3 = SORT ( 2.0/PI ) 

C 4 = 1.0/S OR T ( 2 » 0*P I ) 

SIGN = -1.0 

200 READ( 5*201) (TITLE! I ) ,1=1,7) » NVF >NCASES 

201 FORMAT ( 7 A 1 0 > 2 X » 2 I 4 ! 

I F ( EOF * 5 ) 300* 301 

301 WR I TE ( 6 * 70 ) 

70 FORMAT ( 1H1 *///// /////////) 

WR I T E { 6 * 7 1 ) 

71 FORMAT ( 1H0 »47X » 35HBAYES IAN ABILITY ESTIMATION PROGRAM/53X » 

X2 5H THREE ITEM PARAMETER CASE//////) 

WRITEI 6»72 ) 

72 FORMAT ( 1HU » 5 6X » 17H BUREAU OF TESTING /////) 

WRITE(6,73) 

73 FORMAT ( 1H » 52X » 24HUN I VERS I TY OF WASHINGTON ////////) 

R EAD (5*74) NAME * I TERM » 1USE. EPS I LON » N I TEMS 

74 FORMAT (4A10*22X > I 2 , I 4 , F8 . 4 , I 4 ) 

WRITE (6* 75 ) NAME »N ITEMS 

75 FORMAT ( 1HO » 25X » 10H I T EM BANK » 5X > 4A 10 / // /26X » 1 2HTOTAL ITEMS *4X9 

X I 4 / / / / ) 

I F ( 1 TERM-2 ) 76*77,78 

76 EPS I LON= . 0 
GO TO 79 

77 IUSE*N ITEMS 

78 CONTINUE 

79 WRITE ( 6*80 ) IUSE*EPSILON 

80 FORMAT ( IHO » 2 5X * 20HTERM I NAT I ON CR I TER I A/ /30X » 1 6HNUMBER OF ITEMS ♦ 
XI10//30X-)15HSTANDARD ERROR *3X»F10.4) 

READ (5*31) (A! I ) » 1 = 1 »N ITEMS) 

READ (5,31) (B( I ) » 1 = 1 *N ITEMS) 

READ (5,31/ (C( I) , I = 1 »N I TEMS ) 

31 FORMAT ( 10F8.4 ) 

WRITE (6, 84) 

84 FORMAT (1H1) 

CALL VARFMT ( NVF » FMT ) 

NK = 0 

100 NX = NK + 1 
I PRES a 0 

IF (NK-NCASES) 101*101,200 

101 WRITE (6*28) (TITLE (I)*I=1»7) 

28 FORMAT ( 1H1,30X,7A10////) 

RcAD(5»FMT ) ID, ( IR( I > , I =1 *NI TEMS ) 

ABLE=0.0 
VAR=1 .0 
SEE=SQRT ( VAR ) 

WR I T E ( 6 , 8 1 ) ID, ABLE, VAR 

81 FORMAT ( 1HO»10X»15HEXAMINEE NUMBER » 1 1 0/ / 16X » 1 5HPR I OR EST IMATES//2 IX 
X s 8HAB I L I TY jF10o5/21X»9H VARIANCE ,F9.5//) 

WRITE(6,82 ) w) 



FORMAT ( 1H-',15X» AH ITEM* 2AX*9H ITEM BANK ♦ 2 AX ♦ 7HAB I L ITY » 22X * 
X1AHSTANDARD ERROR / 1 2X , 1 2HPRESENT AT I ON » 2 1 X » 6HNUMBER i 26X , 8HEST I MATE ♦ 
X2. 2 X ♦ 1 1 HOF ESTIMATE//) 

DO 20 I = 1 * N I T EMS 
IELIM(I) = 0 
NLEFT = 0 

IF( IPRES.GE.IUSE) GO TO 100 

I F ( EPS I LON oGE • SEE ) GO TO 100 

DO 13 1 = 1 tN ITEMS 

IF ( I -IEL I M < M ) 12,13 0.2 

NLEFT = NLEFT + 1 

I SUB (NLEFT) = I 

CONTINUE 

I F ( NLEFT) 100* 100 » 14 
DO 10 1 = 1, NLEFT 
ITEM » I SUB ( I ) 

RID s 1.0/(A( ITEM)*A( ITEM) ) + VAR 
D(I) = (31 ITEM) "ABLE)/ SORT ( 2 • 0*R ( I ) ) 

ER( I ) =ERF( D< I ) ) 

SER = l.O-ERC I ) 

CITEM = Cl ITEM) 

C 5 = 1.0-CITEM 
FI = 1.00SER/2.0 ) 

F 2 ={CITEM+(C5/2.0)*SERi*EXP(2.0*D( I)*D( I ) ) 

BETA < I ) = (I,0/C5)*IR(I)/VAR)*F1*F2*{1.0/U.0+CITEM*F1) ) 

RMIN = BET A ( 1 } 

JSUB = 1 

DO 1 J=1»NLEFT 

I F ( RMIN - BET A ( J ) ) 10,2 

RMIN = BET A I J ) 

JSUB = J 
CONTINUE 

ITEM = I SUB ( JSUB ) 

RITEM = R(JSUB) 

DITEM = D I JSUB) 

IELIM (ITEM) = ITEM 
I PRES = I PRES + 1 
TEM = DITEM #D IT EM 
IFUR(ITEM)) 3»3>A 
PER = 1.0+ERIJSUB) 

ABLE = ABLE-C3* ( VAR /SORT (RITEM) ) *EXP t S I GN*TEM) /PER 
FI = C2#( 1.0/ ( 1.0+C 1.0/ { A < ITEM) *A( ITEM) )*< 1.0/VAR) )) ) 

F 2 = 1.0/( (EXP(TEM)*PER)**2) 

F3 = C1+DITEM#EXP(TEM)*PER 
GO TO 5 

SER = UO-ERUSUB) 

CITEM = C( ITEM) 

C 5 = 1.0-CITEM 

FA = ( 1 o 0 / ( C ITE‘M+ (C5 /2« 0 ) #SER ) ) 

A8LE=ABLE+CA*C5*F4*EXP( SJGN*TEM>*( VAR/SQRT< RITEM) ) 

FI = C5*C1*FA 

F 2 = ( VAR/RITEM)*EXP(2«0*SIGN*TEM)/SER 
F 3 = ( <C1-DITEM*EXP( TEM) *SER )-( Cl TEM*C1*FA ) ) 

VAR = VAR *<i.0~Fl*F2*F3) 

SEE=SQRT ( VAR ) 

WRITE (6, 83) IPRES* ITEM* ABLE, SEE 
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FORMAT ( 1HU,15X» 1 4 , 26X » 1 4 » 26X »F8 . 4 » 25X »F8 .4 ) 

GO TO 50 
300 STOP 
END 

FUNCTION ERF(X) 

* ERROR FUNCTION FOR) 0 I X - 3*95, MACCLAURIN SERIES 

* FOR X * 10, ERF I X ) = 1 *0 

* ACCURACY PARAMETER IS *1E-10 
ERF=0*01!F(X.EQ.0.0)RETURN$ES=SIGN( 1 *0 ,X ) $XX=E=Y=A6S (X)SJ=l$IF(Y*l 

XT *3.96)GOT01$ERF=1.0JRETURN 

1 S =F = 1 $D02 I =1 ,6015=5* (~1 ) SF = F* I $ J = J+2$XX=XX*Y*Y 
T = XX/(J*F)$IF(T*LT**1E-10)GOTO'31E = E + T*S 

2 CONTINUE 

3 ERF=ABS(E) *2*0/1 .7724538509053 2* ESSRETURN1END 
SUBROUTINE VARFMT ( NVF ,FMT ) 

C NVF = NUMBER OF VARIABLE FORMAT CARDS 

DIMENSION FMT ( 1 ) 

NVF=NVF*20 

READ ( 5 ,470 ) ( FMT ( I) * I * 1 #NVF ) 

470 FORMAT I 20A4 ) 

WRITE! 6,471) ( FMT ( I ) ,1=1, NVF) 

471 F0RMAT(1H0,7H FORMAT/ ( 10X,2CA4) ) 

RETURN 

END 
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